Information Theory and Statistical Physics

Relationships between information theory and statistical physics have been recognized over the last few decades. One such aspect is identifying structures of optimization problems pertaining to certain information-theoretic settings and drawing analogy to parallel structures arising in statistical physics, and then borrowing statistical mechanical insights, as well as powerful analysis techniques (like the replica method) from statistical physics to the dual informationtheoretic setting of interest. Another aspect is the application of the maximum entropy principle, which emerged in statistical mechanics and treating it as a general guiding principle to problems in information theory e.g. signal processing,speech coding ,spectrum estimation. In the reverse side, we can consider statistical mechanics as a form of statistical reference. Information theory gives us a constructive criterion of setting up probability distributions on the basis of partial knowledge, leading to maximum-entropy estimate. So, the usual rules in statistical physics are an immediate consequence of maximum-entropy principle. The facts about maximization of entropy were stated by Gibbs much early, but this property was were treated as side remarks, not providing any justification for the methods of statistical mechanics [2]. This missing ”feature” has been supplied by information theory. So, now entropy can be taken as a starting point of concept, and the fact that a probability distribution maximizes the entropy subject to certain constraints becomes an essential fact which justifies use of that distribution for inferences. Landauers erasure principle [6] provides a powerful link between information theory and physics. Information processing or even the storage of information leads to entropy.As per the principle,the erasure of every bit of information increases the thermodynamic entropy of the world by k ln 2, where k is Boltzmann’s constant(1.38 x 10−23 J/K), and so, this suggests very strong link between the two areas. The report will look at some of the parallels between these two areas. First it discusses results of [4] where he gives an analogy of the information inequality, and Data Processing Theorem(DPT) with the second law of thermodynamics. DPT is used in many proofs of converses of theorems in information theory, hence the roots of fundamental limits of Information Theory can be attributed to the laws of physics. Then it discusses a result [3] about Chernoff bounds and